Identifying Programmer Ability Using Peer Evaluation: An Exploratory Study

Programmer ability is important to assess in both research and professional settings, but it is very difficult to measure directly. The objective of this study was to measure the ability of programmers to accurately rate the programming ability of their peers. The participants were computer science students in a senior-level undergraduate programming languages course. The peer rating of ability on a 5-point scale was compared with the average programming assignment grade. The Spearman correlation rho value was between .432 and .684 depending on which measure of central tendency was used (mode, median, mean, weighted mean), compared with .48 when using individual assignment data as a predictor. Initial results indicate that peerratings are a promising mechanism for identifying the programming ability of a set of developers. This study leaves many open questions that will be the subject of future studies.